Raised `relativeAccuracy` to `0.2` since `0.1` causes `~16%` random failures at `n=10,000` due to expected `stddev` fluctuations. by AlexeyKuznetsov-DD · Pull Request #9588 · DataDog/dd-trace-java

AlexeyKuznetsov-DD · 2025-09-22T20:45:45Z

What Does This Do

Raised relativeAccuracy to 0.2 since 0.1 causes ~16% random failures at n=10,000 due to expected stddev fluctuations.

Motivation

Green CI.

Additional Notes

Fixed flaky test that failing with 16% probability.

…failures at `n=10,000` due to expected `stddev` fluctuations.

datadog-official · 2025-09-22T21:01:00Z

🎯 Code Coverage
• Patch Coverage: 100.00%
• Total Coverage: 80.34% (+20.51%)

View detailed report

_{This comment will be updated automatically if new data arrives.

🔗 Commit SHA: acd784f | Docs | Was this helpful? Give us feedback!}

pr-commenter · 2025-09-22T21:21:03Z

Benchmarks

Startup

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	alexeyk/fixed-flaky-histogram-test
git_commit_date	1758571586	1758573786
git_commit_sha	`7b1d89d`	`acd784f`
release_version	1.54.0-SNAPSHOT~7b1d89d384	1.54.0-SNAPSHOT~acd784fb1b

See matching parameters

	Baseline	Candidate
application	insecure-bank	insecure-bank
ci_job_date	1758575706	1758575706
ci_job_id	1142107322	1142107322
ci_pipeline_id	77223452	77223452
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version	Linux runner-zfyrx7zua-project-304-concurrent-0-ij1fcn76 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux	Linux runner-zfyrx7zua-project-304-concurrent-0-ij1fcn76 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
module	Agent	Agent
parent	None	None

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 51 metrics, 8 unstable metrics.

Startup time reports for petclinic

gantt
    title petclinic - global startup overhead: candidate=1.54.0-SNAPSHOT~acd784fb1b, baseline=1.54.0-SNAPSHOT~7b1d89d384

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.017 s) : 0, 1017408
Total [baseline] (10.769 s) : 0, 10768769
Agent [candidate] (1.024 s) : 0, 1023972
Total [candidate] (10.775 s) : 0, 10774979
section appsec
Agent [baseline] (1.194 s) : 0, 1194197
Total [baseline] (11.012 s) : 0, 11011837
Agent [candidate] (1.194 s) : 0, 1194402
Total [candidate] (10.938 s) : 0, 10937943
section iast
Agent [baseline] (1.154 s) : 0, 1154148
Total [baseline] (11.028 s) : 0, 11028120
Agent [candidate] (1.152 s) : 0, 1151990
Total [candidate] (11.117 s) : 0, 11117347
section profiling
Agent [baseline] (1.164 s) : 0, 1164463
Total [baseline] (11.038 s) : 0, 11037824
Agent [candidate] (1.162 s) : 0, 1161904
Total [candidate] (11.053 s) : 0, 11053002

baseline results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.017 s	-
Agent	appsec	1.194 s	176.789 ms (17.4%)
Agent	iast	1.154 s	136.74 ms (13.4%)
Agent	profiling	1.164 s	147.055 ms (14.5%)
Total	tracing	10.769 s	-
Total	appsec	11.012 s	243.067 ms (2.3%)
Total	iast	11.028 s	259.351 ms (2.4%)
Total	profiling	11.038 s	269.055 ms (2.5%)

candidate results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.024 s	-
Agent	appsec	1.194 s	170.43 ms (16.6%)
Agent	iast	1.152 s	128.018 ms (12.5%)
Agent	profiling	1.162 s	137.932 ms (13.5%)
Total	tracing	10.775 s	-
Total	appsec	10.938 s	162.964 ms (1.5%)
Total	iast	11.117 s	342.368 ms (3.2%)
Total	profiling	11.053 s	278.023 ms (2.6%)

gantt
    title petclinic - break down per module: candidate=1.54.0-SNAPSHOT~acd784fb1b, baseline=1.54.0-SNAPSHOT~7b1d89d384

    dateFormat X
    axisFormat %s
section tracing
crashtracking [baseline] (1.453 ms) : 0, 1453
crashtracking [candidate] (1.454 ms) : 0, 1454
BytebuddyAgent [baseline] (686.254 ms) : 0, 686254
BytebuddyAgent [candidate] (690.914 ms) : 0, 690914
GlobalTracer [baseline] (257.415 ms) : 0, 257415
GlobalTracer [candidate] (258.957 ms) : 0, 258957
AppSec [baseline] (31.567 ms) : 0, 31567
AppSec [candidate] (31.845 ms) : 0, 31845
Debugger [baseline] (6.37 ms) : 0, 6370
Debugger [candidate] (6.386 ms) : 0, 6386
Remote Config [baseline] (687.066 µs) : 0, 687
Remote Config [candidate] (681.662 µs) : 0, 682
Telemetry [baseline] (12.652 ms) : 0, 12652
Telemetry [candidate] (12.619 ms) : 0, 12619
section appsec
crashtracking [baseline] (1.455 ms) : 0, 1455
crashtracking [candidate] (1.45 ms) : 0, 1450
BytebuddyAgent [baseline] (709.473 ms) : 0, 709473
BytebuddyAgent [candidate] (709.345 ms) : 0, 709345
GlobalTracer [baseline] (249.156 ms) : 0, 249156
GlobalTracer [candidate] (249.754 ms) : 0, 249754
AppSec [baseline] (171.207 ms) : 0, 171207
AppSec [candidate] (171.143 ms) : 0, 171143
Debugger [baseline] (6.065 ms) : 0, 6065
Debugger [candidate] (6.027 ms) : 0, 6027
Remote Config [baseline] (624.564 µs) : 0, 625
Remote Config [candidate] (610.9 µs) : 0, 611
Telemetry [baseline] (9.949 ms) : 0, 9949
Telemetry [candidate] (9.88 ms) : 0, 9880
IAST [baseline] (25.093 ms) : 0, 25093
IAST [candidate] (25.075 ms) : 0, 25075
section iast
crashtracking [baseline] (1.455 ms) : 0, 1455
crashtracking [candidate] (1.456 ms) : 0, 1456
BytebuddyAgent [baseline] (808.711 ms) : 0, 808711
BytebuddyAgent [candidate] (807.473 ms) : 0, 807473
GlobalTracer [baseline] (248.412 ms) : 0, 248412
GlobalTracer [candidate] (248.099 ms) : 0, 248099
AppSec [baseline] (27.553 ms) : 0, 27553
AppSec [candidate] (27.341 ms) : 0, 27341
Debugger [baseline] (6.241 ms) : 0, 6241
Debugger [candidate] (6.188 ms) : 0, 6188
Remote Config [baseline] (607.521 µs) : 0, 608
Remote Config [candidate] (587.276 µs) : 0, 587
Telemetry [baseline] (8.373 ms) : 0, 8373
Telemetry [candidate] (8.138 ms) : 0, 8138
IAST [baseline] (31.77 ms) : 0, 31770
IAST [candidate] (31.729 ms) : 0, 31729
section profiling
ProfilingAgent [baseline] (101.95 ms) : 0, 101950
ProfilingAgent [candidate] (101.567 ms) : 0, 101567
crashtracking [baseline] (1.437 ms) : 0, 1437
crashtracking [candidate] (1.442 ms) : 0, 1442
BytebuddyAgent [baseline] (719.179 ms) : 0, 719179
BytebuddyAgent [candidate] (717.716 ms) : 0, 717716
GlobalTracer [baseline] (235.604 ms) : 0, 235604
GlobalTracer [candidate] (235.038 ms) : 0, 235038
AppSec [baseline] (31.173 ms) : 0, 31173
AppSec [candidate] (31.137 ms) : 0, 31137
Debugger [baseline] (6.511 ms) : 0, 6511
Debugger [candidate] (6.464 ms) : 0, 6464
Remote Config [baseline] (721.403 µs) : 0, 721
Remote Config [candidate] (734.657 µs) : 0, 735
Telemetry [baseline] (16.746 ms) : 0, 16746
Telemetry [candidate] (16.684 ms) : 0, 16684
Profiling [baseline] (102.547 ms) : 0, 102547
Profiling [candidate] (102.157 ms) : 0, 102157

Startup time reports for insecure-bank

gantt
    title insecure-bank - global startup overhead: candidate=1.54.0-SNAPSHOT~acd784fb1b, baseline=1.54.0-SNAPSHOT~7b1d89d384

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.018 s) : 0, 1017855
Total [baseline] (8.679 s) : 0, 8678794
Agent [candidate] (1.018 s) : 0, 1017570
Total [candidate] (8.664 s) : 0, 8664239
section iast
Agent [baseline] (1.161 s) : 0, 1160771
Total [baseline] (9.292 s) : 0, 9292422
Agent [candidate] (1.15 s) : 0, 1150036
Total [candidate] (9.383 s) : 0, 9383035

baseline results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.018 s	-
Agent	iast	1.161 s	142.916 ms (14.0%)
Total	tracing	8.679 s	-
Total	iast	9.292 s	613.628 ms (7.1%)

candidate results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.018 s	-
Agent	iast	1.15 s	132.466 ms (13.0%)
Total	tracing	8.664 s	-
Total	iast	9.383 s	718.797 ms (8.3%)

gantt
    title insecure-bank - break down per module: candidate=1.54.0-SNAPSHOT~acd784fb1b, baseline=1.54.0-SNAPSHOT~7b1d89d384

    dateFormat X
    axisFormat %s
section tracing
crashtracking [baseline] (1.46 ms) : 0, 1460
crashtracking [candidate] (1.438 ms) : 0, 1438
BytebuddyAgent [baseline] (685.389 ms) : 0, 685389
BytebuddyAgent [candidate] (685.82 ms) : 0, 685820
GlobalTracer [baseline] (257.218 ms) : 0, 257218
GlobalTracer [candidate] (257.472 ms) : 0, 257472
AppSec [baseline] (31.62 ms) : 0, 31620
AppSec [candidate] (31.557 ms) : 0, 31557
Debugger [baseline] (6.349 ms) : 0, 6349
Debugger [candidate] (6.312 ms) : 0, 6312
Remote Config [baseline] (686.209 µs) : 0, 686
Remote Config [candidate] (675.704 µs) : 0, 676
Telemetry [baseline] (14.219 ms) : 0, 14219
Telemetry [candidate] (13.369 ms) : 0, 13369
section iast
crashtracking [baseline] (1.482 ms) : 0, 1482
crashtracking [candidate] (1.453 ms) : 0, 1453
BytebuddyAgent [baseline] (814.013 ms) : 0, 814013
BytebuddyAgent [candidate] (806.626 ms) : 0, 806626
GlobalTracer [baseline] (249.404 ms) : 0, 249404
GlobalTracer [candidate] (246.892 ms) : 0, 246892
IAST [baseline] (32.152 ms) : 0, 32152
IAST [candidate] (30.839 ms) : 0, 30839
AppSec [baseline] (27.375 ms) : 0, 27375
AppSec [candidate] (28.262 ms) : 0, 28262
Debugger [baseline] (6.275 ms) : 0, 6275
Debugger [candidate] (6.165 ms) : 0, 6165
Remote Config [baseline] (607.354 µs) : 0, 607
Remote Config [candidate] (602.902 µs) : 0, 603
Telemetry [baseline] (8.398 ms) : 0, 8398
Telemetry [candidate] (8.296 ms) : 0, 8296

Load

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	alexeyk/fixed-flaky-histogram-test
git_commit_date	1758571586	1758573786
git_commit_sha	`7b1d89d`	`acd784f`
release_version	1.54.0-SNAPSHOT~7b1d89d384	1.54.0-SNAPSHOT~acd784fb1b

See matching parameters

	Baseline	Candidate
application	insecure-bank	insecure-bank
ci_job_date	1758575446	1758575446
ci_job_id	1142107323	1142107323
ci_pipeline_id	77223452	77223452
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version	Linux runner-zfyrx7zua-project-304-concurrent-1-0rhk4nth 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux	Linux runner-zfyrx7zua-project-304-concurrent-1-0rhk4nth 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux

Summary

Found 3 performance improvements and 1 performance regressions! Performance is the same for 8 metrics, 12 unstable metrics.

scenario	Δ mean http_req_duration	Δ mean throughput	candidate mean http_req_duration	candidate mean throughput	baseline mean http_req_duration	baseline mean throughput
scenario:load:insecure-bank:tracing:high_load	better [-629.738µs; -392.427µs] or [-7.961%; -4.961%]	unstable [-38.427op/s; +118.302op/s] or [-6.559%; +20.193%]	7.399ms	625.781op/s	7.910ms	585.844op/s
scenario:load:insecure-bank:profiling:high_load	worse [+242.656µs; +554.758µs] or [+2.809%; +6.421%]	unstable [-93.558op/s; +46.683op/s] or [-17.429%; +8.696%]	9.038ms	513.375op/s	8.639ms	536.812op/s
scenario:load:insecure-bank:iast_GLOBAL:high_load	better [-727.354µs; -321.251µs] or [-6.686%; -2.953%]	unstable [-31.562op/s; +74.500op/s] or [-7.385%; +17.431%]	10.354ms	448.875op/s	10.878ms	427.406op/s
scenario:load:insecure-bank:iast:high_load	better [-660.944µs; -315.593µs] or [-6.686%; -3.193%]	unstable [-34.913op/s; +83.475op/s] or [-7.432%; +17.769%]	9.397ms	494.062op/s	9.885ms	469.781op/s

Request duration reports for insecure-bank

gantt
    title insecure-bank - request duration [CI 0.99] : candidate=1.54.0-SNAPSHOT~acd784fb1b, baseline=1.54.0-SNAPSHOT~7b1d89d384
    dateFormat X
    axisFormat %s
section baseline
no_agent (4.368 ms) : 4314, 4422
.   : milestone, 4368,
iast (9.885 ms) : 9719, 10052
.   : milestone, 9885,
iast_FULL (14.202 ms) : 13916, 14487
.   : milestone, 14202,
iast_GLOBAL (10.878 ms) : 10684, 11072
.   : milestone, 10878,
profiling (8.639 ms) : 8509, 8769
.   : milestone, 8639,
tracing (7.91 ms) : 7792, 8028
.   : milestone, 7910,
section candidate
no_agent (4.326 ms) : 4279, 4373
.   : milestone, 4326,
iast (9.397 ms) : 9243, 9551
.   : milestone, 9397,
iast_FULL (14.465 ms) : 14172, 14758
.   : milestone, 14465,
iast_GLOBAL (10.354 ms) : 10171, 10537
.   : milestone, 10354,
profiling (9.038 ms) : 8879, 9196
.   : milestone, 9038,
tracing (7.399 ms) : 7297, 7501
.   : milestone, 7399,

baseline results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	4.368 ms [4.314 ms, 4.422 ms]	-
iast	9.885 ms [9.719 ms, 10.052 ms]	5.517 ms (126.3%)
iast_FULL	14.202 ms [13.916 ms, 14.487 ms]	9.833 ms (225.1%)
iast_GLOBAL	10.878 ms [10.684 ms, 11.072 ms]	6.51 ms (149.0%)
profiling	8.639 ms [8.509 ms, 8.769 ms]	4.271 ms (97.8%)
tracing	7.91 ms [7.792 ms, 8.028 ms]	3.542 ms (81.1%)

candidate results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	4.326 ms [4.279 ms, 4.373 ms]	-
iast	9.397 ms [9.243 ms, 9.551 ms]	5.071 ms (117.2%)
iast_FULL	14.465 ms [14.172 ms, 14.758 ms]	10.139 ms (234.4%)
iast_GLOBAL	10.354 ms [10.171 ms, 10.537 ms]	6.028 ms (139.3%)
profiling	9.038 ms [8.879 ms, 9.196 ms]	4.712 ms (108.9%)
tracing	7.399 ms [7.297 ms, 7.501 ms]	3.073 ms (71.0%)

Request duration reports for petclinic

gantt
    title petclinic - request duration [CI 0.99] : candidate=1.54.0-SNAPSHOT~acd784fb1b, baseline=1.54.0-SNAPSHOT~7b1d89d384
    dateFormat X
    axisFormat %s
section baseline
no_agent (36.681 ms) : 36394, 36968
.   : milestone, 36681,
appsec (48.546 ms) : 48115, 48976
.   : milestone, 48546,
code_origins (43.095 ms) : 42717, 43473
.   : milestone, 43095,
iast (45.259 ms) : 44860, 45659
.   : milestone, 45259,
profiling (48.371 ms) : 47931, 48812
.   : milestone, 48371,
tracing (42.948 ms) : 42579, 43317
.   : milestone, 42948,
section candidate
no_agent (36.249 ms) : 35960, 36539
.   : milestone, 36249,
appsec (48.147 ms) : 47724, 48571
.   : milestone, 48147,
code_origins (43.784 ms) : 43407, 44161
.   : milestone, 43784,
iast (44.89 ms) : 44487, 45293
.   : milestone, 44890,
profiling (48.233 ms) : 47805, 48662
.   : milestone, 48233,
tracing (43.933 ms) : 43559, 44307
.   : milestone, 43933,

baseline results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	36.681 ms [36.394 ms, 36.968 ms]	-
appsec	48.546 ms [48.115 ms, 48.976 ms]	11.864 ms (32.3%)
code_origins	43.095 ms [42.717 ms, 43.473 ms]	6.414 ms (17.5%)
iast	45.259 ms [44.86 ms, 45.659 ms]	8.578 ms (23.4%)
profiling	48.371 ms [47.931 ms, 48.812 ms]	11.69 ms (31.9%)
tracing	42.948 ms [42.579 ms, 43.317 ms]	6.266 ms (17.1%)

candidate results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	36.249 ms [35.96 ms, 36.539 ms]	-
appsec	48.147 ms [47.724 ms, 48.571 ms]	11.898 ms (32.8%)
code_origins	43.784 ms [43.407 ms, 44.161 ms]	7.535 ms (20.8%)
iast	44.89 ms [44.487 ms, 45.293 ms]	8.641 ms (23.8%)
profiling	48.233 ms [47.805 ms, 48.662 ms]	11.984 ms (33.1%)
tracing	43.933 ms [43.559 ms, 44.307 ms]	7.683 ms (21.2%)

Dacapo

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	alexeyk/fixed-flaky-histogram-test
git_commit_date	1758571586	1758573786
git_commit_sha	`7b1d89d`	`acd784f`
release_version	1.54.0-SNAPSHOT~7b1d89d384	1.54.0-SNAPSHOT~acd784fb1b

See matching parameters

	Baseline	Candidate
application	biojava	biojava
ci_job_date	1758575967	1758575967
ci_job_id	1142107324	1142107324
ci_pipeline_id	77223452	77223452
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version	Linux runner-zfyrx7zua-project-304-concurrent-2-hgmey5yu 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux	Linux runner-zfyrx7zua-project-304-concurrent-2-hgmey5yu 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 11 metrics, 1 unstable metrics.

Execution time for tomcat

gantt
    title tomcat - execution time [CI 0.99] : candidate=1.54.0-SNAPSHOT~acd784fb1b, baseline=1.54.0-SNAPSHOT~7b1d89d384
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.471 ms) : 1460, 1483
.   : milestone, 1471,
appsec (3.713 ms) : 3497, 3930
.   : milestone, 3713,
iast (2.193 ms) : 2131, 2256
.   : milestone, 2193,
iast_GLOBAL (2.243 ms) : 2180, 2306
.   : milestone, 2243,
profiling (2.039 ms) : 1989, 2090
.   : milestone, 2039,
tracing (2.026 ms) : 1976, 2076
.   : milestone, 2026,
section candidate
no_agent (1.469 ms) : 1457, 1480
.   : milestone, 1469,
appsec (3.621 ms) : 3408, 3835
.   : milestone, 3621,
iast (2.202 ms) : 2139, 2264
.   : milestone, 2202,
iast_GLOBAL (2.239 ms) : 2176, 2302
.   : milestone, 2239,
profiling (2.033 ms) : 1982, 2083
.   : milestone, 2033,
tracing (2.02 ms) : 1970, 2069
.   : milestone, 2020,

baseline results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	1.471 ms [1.46 ms, 1.483 ms]	-
appsec	3.713 ms [3.497 ms, 3.93 ms]	2.242 ms (152.4%)
iast	2.193 ms [2.131 ms, 2.256 ms]	722.058 µs (49.1%)
iast_GLOBAL	2.243 ms [2.18 ms, 2.306 ms]	771.723 µs (52.5%)
profiling	2.039 ms [1.989 ms, 2.09 ms]	568.086 µs (38.6%)
tracing	2.026 ms [1.976 ms, 2.076 ms]	554.607 µs (37.7%)

candidate results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	1.469 ms [1.457 ms, 1.48 ms]	-
appsec	3.621 ms [3.408 ms, 3.835 ms]	2.153 ms (146.6%)
iast	2.202 ms [2.139 ms, 2.264 ms]	733.087 µs (49.9%)
iast_GLOBAL	2.239 ms [2.176 ms, 2.302 ms]	770.431 µs (52.5%)
profiling	2.033 ms [1.982 ms, 2.083 ms]	564.059 µs (38.4%)
tracing	2.02 ms [1.97 ms, 2.069 ms]	551.048 µs (37.5%)

Execution time for biojava

gantt
    title biojava - execution time [CI 0.99] : candidate=1.54.0-SNAPSHOT~acd784fb1b, baseline=1.54.0-SNAPSHOT~7b1d89d384
    dateFormat X
    axisFormat %s
section baseline
no_agent (15.068 s) : 15068000, 15068000
.   : milestone, 15068000,
appsec (15.037 s) : 15037000, 15037000
.   : milestone, 15037000,
iast (18.56 s) : 18560000, 18560000
.   : milestone, 18560000,
iast_GLOBAL (18.135 s) : 18135000, 18135000
.   : milestone, 18135000,
profiling (15.633 s) : 15633000, 15633000
.   : milestone, 15633000,
tracing (15.071 s) : 15071000, 15071000
.   : milestone, 15071000,
section candidate
no_agent (15.228 s) : 15228000, 15228000
.   : milestone, 15228000,
appsec (15.051 s) : 15051000, 15051000
.   : milestone, 15051000,
iast (18.362 s) : 18362000, 18362000
.   : milestone, 18362000,
iast_GLOBAL (18.001 s) : 18001000, 18001000
.   : milestone, 18001000,
profiling (15.352 s) : 15352000, 15352000
.   : milestone, 15352000,
tracing (15.117 s) : 15117000, 15117000
.   : milestone, 15117000,

baseline results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	15.068 s [15.068 s, 15.068 s]	-
appsec	15.037 s [15.037 s, 15.037 s]	-31.0 ms (-0.2%)
iast	18.56 s [18.56 s, 18.56 s]	3.492 s (23.2%)
iast_GLOBAL	18.135 s [18.135 s, 18.135 s]	3.067 s (20.4%)
profiling	15.633 s [15.633 s, 15.633 s]	565.0 ms (3.7%)
tracing	15.071 s [15.071 s, 15.071 s]	3.0 ms (0.0%)

candidate results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	15.228 s [15.228 s, 15.228 s]	-
appsec	15.051 s [15.051 s, 15.051 s]	-177.0 ms (-1.2%)
iast	18.362 s [18.362 s, 18.362 s]	3.134 s (20.6%)
iast_GLOBAL	18.001 s [18.001 s, 18.001 s]	2.773 s (18.2%)
profiling	15.352 s [15.352 s, 15.352 s]	124.0 ms (0.8%)
tracing	15.117 s [15.117 s, 15.117 s]	-111.0 ms (-0.7%)

bric3

Couple of questions:

Alternatively would it be better to retry ?
I haven't looked deeply into the test, but I noticed increasing relativeAccuracy numbers, is there a pattern to be careful to follow ?

Otherwise, that looks OK to me.

AlexeyKuznetsov-DD · 2025-09-23T14:48:39Z

Couple of questions:

Alternatively would it be better to retry ?

I haven't looked deeply into the test, but I noticed increasing relativeAccuracy numbers, is there a pattern to be careful to follow ?

Otherwise, that looks OK to me.

The test already has retries at the Gradle level. However, this makes it show up as flaky in the CI-Vis dashboard, which is why adding another retry doesn’t really improve things, just make test code a bit complex.
This particular test seems inconsistent to me. Originally it was declared like this:

    normal     |   10000  | [1000D, 10D]    | 0.01
    normal     |   100000 | [1000D, 10D]    | 0.02

The odd part is that the second test uses a larger sample size (100k vs 10k) with otherwise identical parameters, yet its relativeAccuracy is higher. Statistically, larger samples should give more stable results, not less. So having a relativeAccuracy for the smaller sample makes little sense.

bric3 · 2025-09-23T16:36:52Z

Yeah that seemed odd to me for the same reasons you stated, so let's go ahead with this change :)

Raised relativeAccuracy to 0.2 since 0.1 causes `~16% random …

acd784f

…failures at `n=10,000` due to expected `stddev` fluctuations.

AlexeyKuznetsov-DD requested review from PerfectSlayer, mcculls and sarahchen6 September 22, 2025 20:45

AlexeyKuznetsov-DD self-assigned this Sep 22, 2025

AlexeyKuznetsov-DD added comp: testing Testing tag: no release notes Changes to exclude from release notes labels Sep 22, 2025

AlexeyKuznetsov-DD marked this pull request as ready for review September 22, 2025 20:45

AlexeyKuznetsov-DD requested a review from a team as a code owner September 22, 2025 20:45

bric3 reviewed Sep 23, 2025

View reviewed changes

AlexeyKuznetsov-DD requested a review from bric3 September 23, 2025 14:48

sarahchen6 approved these changes Sep 23, 2025

View reviewed changes

bric3 approved these changes Sep 23, 2025

View reviewed changes

AlexeyKuznetsov-DD merged commit 11e677e into master Sep 23, 2025
551 checks passed

AlexeyKuznetsov-DD deleted the alexeyk/fixed-flaky-histogram-test branch September 23, 2025 20:06

github-actions Bot added this to the 1.54.0 milestone Sep 23, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Raised `relativeAccuracy` to `0.2` since `0.1` causes `~16%` random failures at `n=10,000` due to expected `stddev` fluctuations.#9588

Raised `relativeAccuracy` to `0.2` since `0.1` causes `~16%` random failures at `n=10,000` due to expected `stddev` fluctuations.#9588
AlexeyKuznetsov-DD merged 1 commit intomasterfrom
alexeyk/fixed-flaky-histogram-test

AlexeyKuznetsov-DD commented Sep 22, 2025 •

edited

Loading

Uh oh!

datadog-official Bot commented Sep 22, 2025

Uh oh!

pr-commenter Bot commented Sep 22, 2025

Uh oh!

bric3 left a comment

Uh oh!

AlexeyKuznetsov-DD commented Sep 23, 2025

Uh oh!

bric3 commented Sep 23, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

AlexeyKuznetsov-DD commented Sep 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What Does This Do

Motivation

Additional Notes

Uh oh!

datadog-official Bot commented Sep 22, 2025

Uh oh!

pr-commenter Bot commented Sep 22, 2025

Benchmarks

Startup

Parameters

Summary

Load

Parameters

Summary

Dacapo

Parameters

Summary

Uh oh!

bric3 left a comment

Choose a reason for hiding this comment

Uh oh!

AlexeyKuznetsov-DD commented Sep 23, 2025

Uh oh!

bric3 commented Sep 23, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

AlexeyKuznetsov-DD commented Sep 22, 2025 •

edited

Loading